S3Drive - Hide .empty even if Hide dotfiles is disabled

Not sure if this will be that useful for others but It would be great if S3Drive automatically excludes .empty from explorer considering it's just to index a folder and nothing else. But for me and my friends who leverages . for the structure of our folders and files It will be greatly appreciated.

Good idea, we've added this as a feature request: https://s3drive.canny.io/feature-requests/p/add-option-to-hide-empty-file-in-the-listings and shall deliver within next couple weeks. (edited)

1

Hi, this is to let you know that we plan to include this feature in a next release. We've used a new name to differentiate from the: .empty file then hidden .blank files in a folder in a cached and standard view. This is because new .blank file will be encryption compliant, even though it's empty it will contain valid encryption header, so Rclone doesn't fail when processing it. Already created .empty file itself might cause some issues with Sync, we've decided to keep it visible, so affected user can eventually delete it.

Tom

Hi, this is to let you know that we plan to include this feature in a next release. We've used a new name to differentiate from the: .empty file then hidden .blank files in a folder in a cached and standard view. This is because new .blank file will be encryption compliant, even though it's empty it will contain valid encryption header, so Rclone doesn't fail when processing it. Already created .empty file itself might cause some issues with Sync, we've decided to keep it visible, so affected user can eventually delete it.

Great! But about the .empty is there a way to delete them in S3Drive with ease through cache. When searching I just noticed it doesn't have a functionality where you can select all files that would mean you would have to go through them manually.

This is now released in 1.9.8. Please let me know if you find it an improvement. When you create new folder it will be empty, despite containing .blank file. Regarding: .empty cleanup, you could run it with an Rclone command, e.g.: rclone delete s3drive_bucket: --include ".empty", but that would have to be executed on desktop. I have no experience running Rclone as a standalone CLI on Android or iOS. (edited)

To simplify configuring Rclone endpoint in a next version S3Drive will be able to configure (it does that already if e.g. sync is used) and then display name of Rclone remote.

Tom

This is now released in 1.9.8. Please let me know if you find it an improvement. When you create new folder it will be empty, despite containing .blank file. Regarding: .empty cleanup, you could run it with an Rclone command, e.g.: rclone delete s3drive_bucket: --include ".empty", but that would have to be executed on desktop. I have no experience running Rclone as a standalone CLI on Android or iOS. (edited)

Hi Tom yup that's it! Btw is it just me or the version is not released yet? ^^

7:54 AM

As for Rclone on Android I'm not sure if such thing exists XD

I2rys (安全)

Hi Tom yup that's it! Btw is it just me or the version is not released yet? ^^

It should've been released on all platforms. On what platform you can't see the 1.9.8?

I2rys (安全)

As for Rclone on Android I'm not sure if such thing exists XD

Some people install Termux and then Rclone inside.

Tom

It should've been released on all platforms. On what platform you can't see the 1.9.8?

Windows 10

Tom

Some people install Termux and then Rclone inside.

Oh I see, didn't think of that ;-;

I2rys (安全)

Hi Tom yup that's it! Btw is it just me or the version is not released yet? ^^

Sorry, our mistake, we have prepared a release, but haven't approved it. It shall be fixed now.

Tom

Sorry, our mistake, we have prepared a release, but haven't approved it. It shall be fixed now.

No worries, can confirm It's fixed now (marked this thread as resolved).

(edited)

@Tom Btw if I use the command you have given to me and delete .empty from S3Drive managed it returns a The Access Key Id you provided does not exist in our records. error

I2rys (安全)

@Tom Btw if I use the command you have given to me and delete .empty from S3Drive managed it returns a The Access Key Id you provided does not exist in our records. error

Are you using external S3 or managed account? That command won't work for the managed S3 storage that we provide, as we're using internally using different authentication protocol not supported by Rclone. We will either provide a way to generate S3 credentials to managed storage or merge our changes to Rclone (this is unlikely, as this auth is S3Drive specific).

Tom

Are you using external S3 or managed account? That command won't work for the managed S3 storage that we provide, as we're using internally using different authentication protocol not supported by Rclone. We will either provide a way to generate S3 credentials to managed storage or merge our changes to Rclone (this is unlikely, as this auth is S3Drive specific).

I see make sense, yes I was using the managed account. ;-;

Actually, there is a way to connect from Rclone to managed account. When you open rclone.conf you will see config like:

[s3drive_auto_acddd458-d307-4053-b072-1180909eb54a]
type = s3
secret_access_key = applicationKey
access_key_id = keyId
endpoint = https://storage.kapsa.io
jwt_access_token = eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJhdXRoZW50aWNhdGVkIiwiZXhwIjoxNzI0MjM2MTg5LCJpYXQiOjE3MjQyMzI1ODksInN1YiI6ImFjZGRkNDU4LWQzMDctNDA1My1iMDcyLTExODA5MDllYjU0YSIsImVtYWlsIjoiczN1bHRpbWF0ZUB0cjMuZXUiLCJwaG9uZSI6IiIsImFwcF9tZXRhZGF0YSI6eyJwcm92aWRlciI6ImVtYWlsIiwicHJvdmlkZXJzIjpbImVtYWlsIl19LCJ1c2VyX21ldGFkYXRhIjp7ImVtYWlsIjoiczN1bHRpbWF0ZUB0cjMuZXUiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInBob25lX3ZlcmlmaWVkIjpmYWxzZSwic3ViIjoiYWNkZGQ0NTgtZDMwNy00MDUzLWIwNzItMTE4MDkwOWViNTRhIn0sInJvbGUiOiJhdXRoZW50aWNhdGVkIiwiYWFsIjoiYWFsMSIsImFtciI6W3sibWV0aG9kIjoicGFzc3dvcmQiLCJ0aW1lc3RhbXAiOjE3MjQyMzI1ODl9XSwic2Vzc2lvbl9pZCI6ImE1NGU3NThhLTVlNjMtNGEzNi1iMjJmLTQ2NDA0ZmI2MjkwZiIsImlzX2Fub255bW91cyI6ZmFsc2V9.OTC89YZcCMl4gf_9QM6c5fEWyvlXtIabLR9FCtBuqMY
provider = Other
region = us-east-1

[s3drive_s3ultimate]
type = alias
remote = s3drive_auto_acddd458-d307-4053-b072-1180909eb54a:bucket

Normally if you've wanted to e.g. list files using Rclone you would issue command: rclone ls s3drive_s3ultimate: The thing is that authorization token must be included in the request for the managed account and passed using: --header flag. You can use below placeholder and paste the token from the: jwt_access_token field. rclone ls --header "Authorization: Bearer <tokenHere>" s3drive_s3ultimate: (edited)

12:02 PM

The end result would be:

rclone --header "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJhdXRoZW50aWNhdGVkIiwiZXhwIjoxNzI0MjQ1MTAwLCJpYXQiOjE3MjQyNDE1MDAsInN1YiI6ImFjZGRkNDU4LWQzMDctNDA1My1iMDcyLTExODA5MDllYjU0YSIsImVtYWlsIjoiczN1bHRpbWF0ZUB0cjMuZXUiLCJwaG9uZSI6IiIsImFwcF9tZXRhZGF0YSI6eyJwcm92aWRlciI6ImVtYWlsIiwicHJvdmlkZXJzIjpbImVtYWlsIl19LCJ1c2VyX21ldGFkYXRhIjp7ImVtYWlsIjoiczN1bHRpbWF0ZUB0cjMuZXUiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInBob25lX3ZlcmlmaWVkIjpmYWxzZSwic3ViIjoiYWNkZGQ0NTgtZDMwNy00MDUzLWIwNzItMTE4MDkwOWViNTRhIn0sInJvbGUiOiJhdXRoZW50aWNhdGVkIiwiYWFsIjoiYWFsMSIsImFtciI6W3sibWV0aG9kIjoicGFzc3dvcmQiLCJ0aW1lc3RhbXAiOjE3MjQyNDE1MDB9XSwic2Vzc2lvbl9pZCI6ImRmZDEzZGQwLTFlZjUtNDUzZi04MDVjLWI3NmUzMzcyNTU2NiIsImlzX2Fub255bW91cyI6ZmFsc2V9.czC4IcDgcFJy2PAXocFECD-PYcJWUgT_5ikDLJhqXWk" ls s3drive_s3ultimate:

and this would allow you to perform operations on the managed account using Rclone. Caveat: Token will work for up to an hour, so you would need to repeat this pretty soon. In the future we will either provide S3 credentials that can be used or token that doesn't expire or provide better way of storing these settings in Rclone permanently: https://forum.rclone.org/t/storing-config-flag-in-the-back-end-configuration-e-g-store-header-permanently-on-the-back-end-level/47390 (edited)

Tom

Actually, there is a way to connect from Rclone to managed account. When you open rclone.conf you will see config like:

[s3drive_auto_acddd458-d307-4053-b072-1180909eb54a]
type = s3
secret_access_key = applicationKey
access_key_id = keyId
endpoint = https://storage.kapsa.io
jwt_access_token = eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJhdXRoZW50aWNhdGVkIiwiZXhwIjoxNzI0MjM2MTg5LCJpYXQiOjE3MjQyMzI1ODksInN1YiI6ImFjZGRkNDU4LWQzMDctNDA1My1iMDcyLTExODA5MDllYjU0YSIsImVtYWlsIjoiczN1bHRpbWF0ZUB0cjMuZXUiLCJwaG9uZSI6IiIsImFwcF9tZXRhZGF0YSI6eyJwcm92aWRlciI6ImVtYWlsIiwicHJvdmlkZXJzIjpbImVtYWlsIl19LCJ1c2VyX21ldGFkYXRhIjp7ImVtYWlsIjoiczN1bHRpbWF0ZUB0cjMuZXUiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInBob25lX3ZlcmlmaWVkIjpmYWxzZSwic3ViIjoiYWNkZGQ0NTgtZDMwNy00MDUzLWIwNzItMTE4MDkwOWViNTRhIn0sInJvbGUiOiJhdXRoZW50aWNhdGVkIiwiYWFsIjoiYWFsMSIsImFtciI6W3sibWV0aG9kIjoicGFzc3dvcmQiLCJ0aW1lc3RhbXAiOjE3MjQyMzI1ODl9XSwic2Vzc2lvbl9pZCI6ImE1NGU3NThhLTVlNjMtNGEzNi1iMjJmLTQ2NDA0ZmI2MjkwZiIsImlzX2Fub255bW91cyI6ZmFsc2V9.OTC89YZcCMl4gf_9QM6c5fEWyvlXtIabLR9FCtBuqMY
provider = Other
region = us-east-1

[s3drive_s3ultimate]
type = alias
remote = s3drive_auto_acddd458-d307-4053-b072-1180909eb54a:bucket

Normally if you've wanted to e.g. list files using Rclone you would issue command: rclone ls s3drive_s3ultimate: The thing is that authorization token must be included in the request for the managed account and passed using: --header flag. You can use below placeholder and paste the token from the: jwt_access_token field. rclone ls --header "Authorization: Bearer <tokenHere>" s3drive_s3ultimate: (edited)

Hi, I see thank you so much also can't wait for S3Drive new update too, speaking of that where can I get the jwt_access_token is it from the web login?

Pretty much. This token gets updated in rclone.conf once renewed (every hour) on desktop/mobile, but you can also look it up in the browser inspector if you open our web client.

1

Hi @I2rys (安全), just to give you an update. With recent update, the jwt_access_token isn't needed for managed accounts. Rclone config will simply contain valid S3 access_key_id and secret_access_key values. Please note that these will change every 3600s, but we plan changes in that area as well in the future, so they're more persistent.

Tom

Hi @I2rys (安全), just to give you an update. With recent update, the jwt_access_token isn't needed for managed accounts. Rclone config will simply contain valid S3 access_key_id and secret_access_key values. Please note that these will change every 3600s, but we plan changes in that area as well in the future, so they're more persistent.

Hello, thank you for the heads up! Can you please elaborate Rclone config will simply contain valid S3 access_key_id and secret_access_key values. on where I can find it or is it the same process as getting the jwt_access_token?

This config is maintained by the S3Drive app once you're logged in and app is opened. Alternatively, you can enforce Rclone config generation using this setting:

Tom

This config is maintained by the S3Drive app once you're logged in and app is opened. Alternatively, you can enforce Rclone config generation using this setting:

I'm using the newest version of S3Drive and I can't seem to find Rclone remote for the managed account. Perhaps I'm doing it wrong. (edited)

Sorry, I've misguided you. On managed account this should happen automatically, at the moment around every 3600s when the new temp S3 credentials are generated they will end up in your Rclone config.

10:43 AM

My whole point is, that the Rclone config that S3Drive generates now (since recent update) can be used directly in the CLI without any additional flags e.g.: --header "Authorization: Bearer <tokenHere>" as previously. (edited)

1

No worries and I understand now. Btw how are they generated because I look at the rclone config I can't seem to find the managed account temp crdentials.

I2rys (安全)

No worries and I understand now. Btw how are they generated because I look at the rclone config I can't seem to find the managed account temp crdentials.

They would look similarly to this:

[s3drive_auto_9b4df9c3-63cd-40e8-90fd-4638412edecb]
type = s3
provider = Other
region = us-east-1
secret_access_key = t6rzOIzwAornMjQORsFNCg
access_key_id = kapsa-8bae97c2-3f05-41f4-bf77-447240583504
endpoint = https://storage.kapsa.io

Tom

They would look similarly to this:

[s3drive_auto_9b4df9c3-63cd-40e8-90fd-4638412edecb]
type = s3
provider = Other
region = us-east-1
secret_access_key = t6rzOIzwAornMjQORsFNCg
access_key_id = kapsa-8bae97c2-3f05-41f4-bf77-447240583504
endpoint = https://storage.kapsa.io

Oh my bad I thought it's in the [s3drive_managed] thingy but rather It's in the auto. Thanks!